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Claims 

I claim: 

1. A method for recovering non-rigid 3D shape and motion of an object from a 
video of the object, comprising: 

determining correspondences of a set of features on the object in each image 
of the video; 

factoring the correspondences into a motion matrix and a shape matrix; and 
extracting a 3D model from the factored motion matrix and the shape matrix 
that conforms to the object in the video, the 3D model including a linear basis for 
deformable shape of the object in the video, and for each image a 3D rotations 
matrix, deformation coefficients, and translation vectors. 

2. The method of claim 1 wherein the 3D model has a minimum residual error and 
minimal deformation coefficients. 

3. The method of claim 1 further comprising: 

identifying the set of features in a reference frame of the video; 

determining a location for each feature in the reference frame; 

determining a corresponding location of each feature in each other frame of 
the video, the location of each feature in the reference frame and the locations of 
the corresponding features in each other frame forming the correspondences. 

3. The method of claim 1 further comprising: 

selecting a first image of the video as the reference frame. 
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4. The method of claim 1 further comprising: 

selecting a middle image of the video as the reference frame. 

5. The method of claim 1 further comprising: 

determining a location of each feature in each frame of the video, and 
gradients of image intensities in regions around each feature by comparing to the 
corresponding region in the reference frame. 

6. The method of claim 1 wherein each region is approximately 5x5 pixels 
centered on each feature. 

7. The method of claim 5 wherein the correspondences between successive images 
of the video are constrained to be consistent with a range of possible projections of 
a non-rigid 3D surface of the object having a finite number of deformations to 
enable the determining of the correspondences when regions within the images 
lack identifying marks and texture. 

8. The method of claim 1 wherein the correspondence include quantification of 
spatial and temporal variations of image intensities around each feature, the 
quantification determined from gradients of the image intensities. 

9. The method of claim 8 wherein the spatial variation at any location is X, the 
temporal variation between that location and a corresponding location in another 
image is Y, and X is an estimate of a covariance of an uncertainty in the temporal 
variations Y. 

10. The method of claim 9 wherein the factoring uses a covariance of the spatial 
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variations X to decompose the temporal variations Y into low-dimensional factors 
of the correspondences that preserve information about the temporal variations Y 
in a finite number of dimensions, and a distribution of the uncertainty in the low- 
dimensional factors is identically independently Gaussian, regardless of 
correlational structure of noise and error in the correspondences. 

11. The method of claim 1 further comprising: 

generating a novel video from the video by changing any combination of the 
shape matrix, the 3D rotations matrix, the deformation coefficients, and the 
translation vectors of the 3D model. 

12. The method of claim 6 wherein the constraints of the factoring enforces the 
range of possible projections of the non-rigid 3D surface having the finite number 
of deformations to produce the motion matrix and the shape matrix. 

13. The method of claim 12 further comprising: 

transforming the motion matrix and the shape matrix to be consistent with 
image formation governing 2D projections of the non-rigid 3D object. 

14. A method for recovering a 3D model of non-rigid 3D shape and motion of an 
object directly from an input video of the object, comprising: 

identifying a set of features on the object in a reference image of the input 

video; 

determining correspondences between the set of features in the reference 
image and corresponding features in each other image of the input video; 

factoring the correspondences into a motion matrix and a shape matrix; 
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extracting the 3D model from the factored motion matrix and the shape 
matrix, the 3D model including a linear basis for deformable shape of the object in 
the input video, and for each image a 3D rotations matrix, deformation 
coefficients, and translation vectors; and 

manipulating the 3D model and the input video to generate a novel video 
from the input video. 

15. A system for recovering a non-rigid 3D model of a scene, 

a camera configured to acquire an input video of the scene; 
means for identifying a set of features in the scene in a reference image of 
the input video; 

means for determining correspondences between the set of features in the 
reference image and corresponding features in each other image of the input video; 

means for factoring the correspondences into a motion matrix and a shape 
matrix; 

means for extracting the non-rigid 3D model from the factored motion 
matrix and the shape matrix, the 3D model including a linear basis for deformable 
shape in the scene, and for each image a 3D rotations matrix, deformation 
coefficients, and translation vectors. 

16. The system of claim 15 further comprising: 

a display device configured to display a novel video generated by 
manipulating the non-rigid 3D model and the input video. 

17. The system of claim 15 wherein the scene includes a non-rigid deformable 
object. 
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18. The system of claim 15 further comprising: 

a pointing device 106 to identify the set of features in the reference image. 

19. The system of claim 15 further comprising: 

a computerized vision system to automatically identify the set of features in 
the reference image. 
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